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Chapter  1 
Introduction 


Purpose. 

The  U.S.  Army  Training  and  Doctrine  Command  (TRADOC)  Analysis  Center  (TRAC)  faces 
critical  issues  while  supporting  requirements  generation  and  acquisition  studies  for  the 
Department  of  the  Army  (DA)  and  the  Department  of  Defense.  Decision  makers  are  asking: 

•  How  much  military  capability  is  good  enough? 

•  Is  there  a  capability  gap? 

•  What  is  the  operational  impact  of  the  capability  gap? 

•  Is  the  capability  gap  mitigated? 

•  If  so,  by  how  much? 

To  address  these  questions,  TRAC  developed  a  gap  assessment  approach  that  blends  warfighter 
experience  with  science.  This  paper  describes  the  process,  challenges,  and  opportunities 
associated  with  this  technique. 

Key  Terms. 

•  capability  gap  (or  gap)  -  The  inability  to  execute  a  specified  course  of  action.  The  gap 
may  be  the  result  of  no  existing  capability,  a  lack  of  proficiency  or  sufficiency  in  an 
existing  capability  solution,  or  the  need  to  replace  an  existing  capability  solution  to 
prevent  a  future  gap.1 

•  alternatives  -  Solutions  that  mitigate  gaps. 

•  attribute  -A  quantifiable  system  characteristic.  In  this  case  it  represents  the  system 
characteristics  that  enable  measurement  of  capability  gaps. 

•  metric  -  The  standard  or  range  of  feasible  values  associated  with  an  attribute. 

•  qualitative  gap  -  A  capability  gap  that  does  not  have  a  quantifiable  attribute,  or  has  a 
quantifiable  attribute  in  which  technical,  performance,  or  operational  study  data  are 
unavailable. 

•  quantitative  gap  -  A  capability  gap  that  has  a  quantifiable  attribute  and  associated 
technical,  performance,  or  operational  study  data  available  for  the  comparison  of 
alternatives. 

•  mission  impact  scale  -  Developed  for  each  quantifiable  attribute,  the  mission  impact 
scale  takes  warfighter  inputs  to  establish  severity  of  impact  to  missions  for  the  feasible 
values  of  the  associated  attribute. 

•  alternative  assessment  scale  -  The  results  of  taking  a  mission  impact  scale  and  placing 
the  appropriate  technical,  performance,  or  operational  performance  data  on  the  scale  for 
each  alternative. 


1  Chairman  of  the  Joint  Chiefs  of  Staff  Instruction  3170.01H,  Joint  Capabilities  Integration  and  Development 
System.  10  January  2012. 
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•  gap  assessment  scale  -  A  normalized  scale  across  all  attributes  associated  with  a  single 

gap- 

•  overall  gap  mitigation  scale  -  A  roll-up  of  all  gap  assessment  scales  with  normalized 
input  for  each  study  alternative. 

•  “good  enough”  -  A  point  at  which  warfighters  assess  that  they  can  sufficiently  complete 
their  mission  without  significant  risk. 

•  “optimal”  -  A  warfighter’s  ideal  operating  value  that  allows  completion  of  the  mission 
without  setbacks  or  hardships. 

Background. 

TRAC  has  applied  many  methods  to  assess  capability  gaps.  These  methods  have  been  grounded 
in  various  uses  of  Army  Field  Manual  (FM)  5-19,  Composite  Risk  Management  (CRM).  These 
CRM  methods  have  been  used  to  assess  capability  gaps’  operational  risk  and/or  risk  mitigation 
using  a  likelihood  and  severity  assessment.  Most  of  these  methods,  however,  have  been 
qualitative  in  nature,  purely  based  in  warfighter  opinion.  As  a  qualitative  assessment,  these 
previous  methods  provide  potential  for  warfighters  to  support  new  and  emerging  technologies 
and  systems  without  regard  for  what  is  sufficient  in  an  operational  environment. 

As  funding  constraints  loom,  this  new  gap  assessment  approach  provides  a  mathematical  method 
grounded  in  warfighter  experience.  This  new  approach  may  be  used  to  1)  address  key  decision 
maker  issues,  2)  identify  trade  space  for  requirements,  and  3)  either  help  develop  or  corroborate 
system  requirements  for  capability  documents. 

The  method  that  this  technical  report  describes  was  first  implemented  in  the  Ground  Combat 
Vehicle  (GCV)  Analysis  of  Alternatives  (AoA)  Milestone  B  analysis  in  2012. 

Relevant  Applications. 

•  Provide  a  foundation  to  identify  how  much  capability  is  “good  enough”  to  drive 
requirements  development  and  may  be  implemented  at  multiple  points  throughout  the 
Joint  Capabilities  Integration  and  Development  System  (JCIDS)  and  Defense  Acquisition 
System.  Note  that  the  effect  of  the  approach  is  greater  and  more  beneficial  the  earlier  it  is 
implemented  in  the  JCIDS/acquisition  process. 

•  Corroborate  or  help  refine  capability  requirements. 

•  Identify  potential  trade  space  in  requirements  and  capability  attributes. 

•  Assess  the  operational  impact  or  operational  risk  of  capability  gaps  and  associated 
attributes. 

•  Identify  to  what  degree  potential  solutions  mitigate  the  gap. 

Constraints,  Limitations,  Assumptions. 

Constraints. 

The  resources  available  to  execute  the  analysis  constrain  this  gap  assessment.  Study  time 
constraints  led  to  the  development  of  this  approach,  because  most  of  the  work  can  be 
accomplished  before  data  availability  on  the  potential  gap  solutions  or  study  alternatives. 
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Availability  of  approved  and  refined  gaps  can  also  constrain  this  process.  Regardless  of  the  study 
and  the  robustness  of  the  capability  gaps,  refinement  is  always  required  to  focus  those  capability 
gaps  on  the  measurable  attributes  of  the  potential  solutions  or  study  alternatives  being  compared 
for  gap  mitigation. 

Limitations. 

When  implementing  this  approach  for  a  specific  subject  or  study,  the  data  and  expert/warfighter 
availability  will  be  limited. 

Assumptions. 

The  reader  has  a  basic  understanding  of  JCIDS,  the  Defense  Acquisition  System,  and  TRADOC 
organizational  structure. 

Approach. 

This  new  approach  to  gap  assessment  blends  warfighter  expertise  with  quantifiable  analysis 
through  a  scientific  approach  (figure  1).  The  most  critical  step,  always  executed  at  the  start  of  the 
analysis,  is  identifying  and  refining  the  capability  gaps.  Capability  gaps  must  be  refined  to 
contain  as  much  detail  as  possible,  which  provides  the  basis  for  a  more  quantitative  assessment. 

Identifying  data  sources  for  the  gaps  is  accomplished  in  a  measurement  space2 3  workshop. 

An  opportunity  follows  to  survey  experienced  warfighters  about  mission  impacts  of  the  gaps.  For 
qualitative  gaps,  warfighters  may  be  asked  many  questions.  For  gaps  assessed  through  the  new 
quantitative  approach,  warfighters  are  asked  to  identify  the  mission  impact  for  the  measureable 
attributes  of  each  capability  gap.  A  mathematical  approach  is  then  applied  to  combine  warfighter 
responses  into  a  single  mission  impact  scale  for  each  capability  gap. 

When  paired  with  technical,  performance,  and  operational  effectiveness  data,  the  mission  impact 
scales  enable  the  analyst  to  quantifiably  assess  gap  mitigation  of  potential  alternatives  against  the 
current  capability.  All  quantifiable  gaps  may  then  be  rolled  up  into  a  single  scale  to  identify  the 
alternative  that  best  mitigates  the  gap. 

Sensitivity  analysis  may  be  performed  on  the  quantitative  gaps.  This  may  be  followed  by 
integration  with  the  qualitative  gap  assessment  and  other  study  results,  such  as  operational 
effectiveness  analyses. 

Organization.  Figure  1  depicts  the  organization  of  this  report.  Following  this  introduction, 
Chapter  2,  Gap  Refinement  and  Measurement  Space,  describes  the  challenging  process  of 
identifying  and  refining  the  capability  gaps  requiring  analysis.  This  may  be  an  iterative  process 
with  the  development  of  the  gaps  measurement  space.  Regarding  creation  of  the  warfighting 
foundation  for  the  gap  assessment,  chapter  3,  Warfighter  Workshop,  describes  how  to  get  the 
most  out  of  the  time  with  the  warfighter  to  assess  gaps  both  qualitatively  and  quantitatively. 
These  assessments  are  then  described  separately  in  chapter  4,  Gap  Analysis.  Chapter  5,  Overall 
Assessment  and  Integration,  discusses  how  to  take  the  quantitative  gap  assessments  and  roll  them 
into  a  single  gap  mitigation  scale.  The  chapter  also  presents  sensitivities  that  may  be  performed 


2  TRAC-H-TM- 12-034,  Measurement  Space  Code  of  Best  Practice.  TRADOC  Analysis  Center.  June  2012. 

3 


on  the  quantitative  assessment  and  describes  how  to  integrate  the  qualitative  and  quantitative  gap 
assessments  into  the  overarching  analyses.  Chapter  6’s  Application  Challenges  paragraph 
addresses  the  challenges  in  implementing  this  approach. 


Figure  1.  Gap  Assessment  Approach. 
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Chapter  2 

Gap  Refinement  and  Measurement  Space 


Purpose. 

The  most  challenging  step  in  gap  assessment  is  the  identification  and  refinement  of  the  capability 
gap  statements.  Capability  development  for  the  Army  is  the  responsibility  of  the  Army 
Capabilities  Integration  Center  (ARCIC).  The  capability  needs  analysis  (CNA)  process  ARCIC 
uses,  as  an  integrating  agency,  is  intended  to  become  the  primary  source  for  organizational  gaps. 
At  the  time  of  this  report,  TRADOC  Centers  of  Excellence  (CoEs)  are  the  appropriate  initial 
source  for  capability  gaps. 

This  chapter  describes  the  potential  gap  sources,  how  to  identify  the  critical  elements  of  a  gap, 
how  to  refine  a  gap  to  a  level  needed  for  assessment,  how  to  ensure  gap  measurability  in 
conjunction  with  a  measurement  space  workshop,  and  the  importance  of  attaining  approval  of  the 
refined  gaps  before  their  assessment. 

Capability  Gap  Sources. 

ARCIC  and  TRADOC  CoEs  are  the  official  sources  for  capability  gaps.  Start  your  search  for 
applicable  gaps  through  the  appropriate  TRADOC  CoE.  Official  documents  describing  the 
capability  gaps  are  the  best  sources,  such  as  capability-based  assessments  or  initial  capabilities 
documents  (ICDs).  If  the  capability,  materiel  item,  or  subject  under  study  is  unavailable  from  a 
TRADOC  CoE,  identified  experts  must  develop  the  capability  gaps  before  the  start  of  the  gap 
assessment.  Analysts  should  never  develop  the  gaps.  They  work  with  the  appropriate  CoE  or 
TRADOC/ARCIC  directorate  to  identify  existing  capability  gaps  for  refinement. 

Detail  Required  in  a  Capability  Gap. 

Gap  refinement  workshops  conducted  with  experts  from  the  CoE  are  critical  to  ensuring  detail  is 
captured  in  the  capability  gap  statements.  Even  if  the  gaps  are  recently  created  or  updated 
through  the  CNA  process,  and  even  if  the  capability  gaps  meet  the  criteria  described  below, 
specifics  to  the  potential  solutions  being  studied  will  make  the  capability  gap  refinement  critical. 
Never  skip  this  step. 

Capability  gaps  must: 

•  Specifically  state  the  problem  and  the  associated  task. 

•  Identify  what  conditions  cause  the  problem  and  how  frequently  the  conditions  occur. 

•  Describe  the  gap’s  measurable  component  (a  standard  that  describes  how 
closure/mitigation  of  the  gap  can  be  measured). 

•  Describe  the  operational  or  tactical  risk  associated  with  the  gap. 

If  time  permits  after  the  workshops,  taking  the  refined  gap  statements  to  warfighters  with 
experience  on  the  system  or  capability  under  study  can  corroborate  gaps  and  develop  more  detail 
to  attain  measurability. 
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Threat  Assessment. 

Become  familiar  with  the  System  Threat  Assessment  Report  (STAR)  associated  with  the 
capability  under  study.  This  document  and  additional  threat  assessments  identifying  future 
proliferation  and  worldwide  availability  are  often  key  components  of  the  capability  gaps.  Ensure 
that  details  from  the  STAR  and  other  threat  assessments  are  incorporated  into  the  refined  gap 
statements.  Have  your  threat  representative  document  the  future  proliferation  and  worldwide 
availability  as  an  indicator  of  frequency  of  the  threat. 

Capability  Gap  Measurement  Space. 

Ensure  the  capability  gaps  are  identified  and  understood  before  the  study’s  measurement  space 
workshop.  The  workshop  is  an  opportunity  to  identify  and  develop  the  measureable  components 
or  standards  for  each  gap.  In  this  document,  the  components/standards  are  called  attributes.  The 
quantifiable  system  characteristics  (attributes)  must  be  measurable  (to  a  standard)  and  must  have 
a  valid  data  source  to  execute  this  gap  assessment  method.  The  workshop  provides  the  expertise 
and  process  for  these  gap  details.  It  also  focuses  the  analytic  planning  toward  measuring  gap 
mitigation. 

Gap  Approval. 

Following  the  measurement  space  workshop  with  another  refinement  workshop  may  be 
necessary  to  clarify  additional  questions.  This  refinement  workshop  provides  focus  for  the  CoE 
to  approve  the  gaps  at  the  colonel  level.  Unless  the  study  is  very-high  visibility  (to  the  Chief  of 
Staff  of  the  Army),  colonel-level  approval  at  the  CoE  is  sufficient.  However,  additional  study 
approval  for  the  gaps  is  necessary.  The  study  advisory  group  (SAG)  needs  to  review  and  approve 
the  gaps  and  be  aware  of  their  original  source  and  additional  refinement.  Skipping  SAG  approval 
could  hurt  the  gap  assessment.  See  the  chapter  6  Application  Challenges  paragraph  on  gap 
sources  and  approval. 

Gap  Refinement  and  Measurement  Space  Results. 

Upon  completion  of  the  gap  refinement  process  and  measurement  space  workshop,  the  resulting 
gaps  will  fall  under  two  categories  for  further  analysis:  qualitative  or  quantitative. 

Qualitative  Capability  Gaps. 

Qualitative  gaps  will  describe  a  capability  with  attributes  (measures  and  standards)  that  are  not 
quantitative  or  do  not  have  a  study  data  source  from  which  to  compare  alternatives.  The  language 
will  not  specifically  point  to  a  quantifiable  deficiency  where  technical,  performance,  or 
operational  data  are  available  to  compare  alternatives.  The  gap  must,  therefore,  be  assessed 
qualitatively  by  subject  matter  experts/warfighters  through  a  survey  or  discussion  method. 

For  example,  a  capability  gap  describing  the  inability  of  multiple  platforms  to  perform 
cooperative  engagements  through  visual,  verbal,  and  data  communication  is  qualitative.  The 
potential  exists  to  measure  cooperative  actions  in  an  operational  environment  through  time 
standards  or  “success”  criteria,  such  as  the  quantity  of  threats  killed  due  to  cooperative 
engagements.  However,  the  measurement  space  workshop  may  inform  the  gap  analyst  that  the 
modeling  and  simulations  recommended  for  the  study  do  not  provide  that  level  of  detail.  As  a 
result,  the  gap’s  impact  to  the  mission  of  not  being  able  to  cooperatively  engage  a  target  is 
assessable  only  through  qualitative  assessments. 
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Quantitative  Capability  Gaps. 

When  the  gaps  are  quantitative,  numbers  or  values  represent  the  attribute  and  a  source  exists  for 
those  values  through  technical,  performance,  or  operational  analysis.  For  example,  a  gap 
describing  the  inability  to  travel  fast  enough  in  cross-country  terrain  has  an  obvious  metric  of 
miles  per  hour.  Some,  and  preferably  most,  capability  gaps  are  quantifiable. 
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Chapter  3 

Warfighter  Workshop 


Purpose. 

The  warfighter  workshop  represents  the  venue  in  which  the  study  team  can  collect  feedback  and 
get  an  assessment  of  the  capability  gaps  directly  from  the  actual  users:  warfighters.  The 
workshop  enables  the  study  team  to  distribute  prepared  surveys  and  discussion  questions  to 
which  the  warfighters  can  respond  and  influence  the  analysis.  Success  hinges  on  getting  the  right 
people  with  the  right  experience  to  complete  the  gap  assessments. 

Workshop  Planning. 

Following  the  study  measurement  space  workshop,  the  study  team  will  know  which  capability 
gaps  are  qualitative  and  quantitative  in  nature.  The  measurement  space  workshop  should  also 
give  the  study  team  the  attributes  and  metrics  for  each  quantitative  gap.  The  team  then  begins  the 
planning  for  a  warfighter  gap  assessment  workshop. 

Availability  of  Warfighters. 

The  most  important  element  of  workshop  planning  is  identifying  warfighters  with  the  right 
experience  for  the  study.  The  next  most-critical  element  is  accessing  the  qualified  warfighters  in 
a  timely  fashion.  Consideration  of  the  number  of  warfighters  required  for  defendable  assessment 
results  is  important.  Statistically,  more  than  30  data  points  for  each  question  will  allow  insights 
from  responses  to  be  significant.  Given  that  there  may  be  some  outlier  responses,  providing 
surveys  to  more  than  40  warfighters  should  afford  the  study  team  defendable  results.  More  than 
one  warfighter  workshop  may  be  necessary  because  of  availability  and  location  of  these  experts. 

There  may  also  be  more  than  one  reason  to  schedule  time  with  the  warfighters,  as  discussed  in 
chapter  2.  You  may  want  their  expertise  in  reviewing  the  capability  gaps,  but  at  a  minimum,  you 
need  their  input  for  the  gap  assessment.  The  rest  of  this  chapter  focuses  on  the  planning  and 
survey  development  for  the  gap  assessment  warfighter  workshops. 

Defining  the  Severity  Scales. 

As  chapter  1  noted,  this  method  is  grounded  in  the  mission  impact  severity  ratings  from  FM 
5-19.  The  severity  ratings  are  “negligible,”  “marginal,”  “critical,”  and  “catastrophic.”  These 
terms  and  their  meanings  are  familiar  to  warfighters.  Using  the  FM  5-19  categories  affords  the 
study  team  a  method  to  ensure  warfighters  are  assessing  each  capability  gap  with  a  consistent 
understanding  of  what  each  severity  rating  means  in  relation  to  the  gap. 

Development  of  Surveys. 

Start  the  survey  development  process  with  brainstorming  to  develop  questions  that  can  be  asked 
of  warfighters  to  assess  the  attributes  coming  from  the  measurement  space  workshop.  Questions 
for  the  qualitative  gaps  should  also  be  brainstormed  so  that  the  survey  accounts  for  all  capability 
gaps.  Developing  questions  and  surveys  is  the  most  crucial  element  of  the  entire  gap  assessment 
process,  because  poor  questions  will  lead  to  insignificant  results. 
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Some  capability  gaps  will  have  multiple  gap  attributes  associated  with  them,  and  each  attribute 
will  need  a  separate  question.  For  example,  the  capability  gap  may  be  related  to  mobility,  and  the 
mobility  attributes  are  cross-country  speed,  hard-surface  speed,  and  dash  speed.  Once 
brainstorming  is  complete,  questions  should  be  pared  and  refined  to  be  clear,  concise,  and 
straightforward.  Testing  the  questions  on  military  operations  research  analysts  external  to  the 
study  team  can  help  ensure  the  questions  are  clear  and  answerable.  Testing  also  allows  the  study 
team  to  get  sample  output  and  determine  whether  warfighters  will  interpret  the  questions  as 
expected.  These  questions  will  then  serve  as  the  basis  for  the  surveys  warfighters  complete 
during  the  gap  assessment  workshop.  Not  all  quantitative  gaps  may  be  measured  by  the 
warfighter  and  will  require  technical  expertise.  Select  the  appropriate  experts  for  the  attribute  in 
question  and  use  the  same  techniques  that  follow.  Chapter  6  shows  a  technical  scale  used  in  the 
same  manner  as  the  mission  impact  scales. 

Once  the  survey  questions  are  clearly  identified,  analysts  must  determine  the  nature  of  the 
quantifiable  gap  questions.  Mathematically  speaking,  a  measurement  can  be  continuous  (e.g., 
time  or  miles  per  hour)  or  discrete  (e.g.,  number  of  vehicles)  in  nature.  Each  gap  may  have 
multiple  metrics  that  may  be  purely  continuous,  purely  discrete,  or  a  mix  of  both.  Therein  lies  the 
power  of  this  gap  assessment  approach,  as  it  provides  the  means  to  combine  not  only  multiple 
continuous  scales  of  varying  metrics,  but  also  attribute  scales  of  varying  types. 

After  questions  for  warfighters  are  developed  and  refined,  the  study  team  should  begin  creating 
the  formal  survey.  For  the  attribute  questions  that  are  answerable  on  a  continuous  scale  (e.g., 
time  or  miles  per  hour),  the  range  of  metrics  that  create  the  attribute  scales  should  be  reduced  to 
only  what  is  feasible  for  the  alternatives  under  study.  For  example,  the  feasible  miles  per  hour 
range  for  a  combat  vehicle  probably  does  not  exceed  60  or  70.  An  assessment  of  the  alternative 
capabilities  and  draft  requirement  may  assist  in  setting  the  range. 

The  survey  tool  should  ask  warfighters  to  identify  the  mission  impact  severity3  (“catastrophic,” 
“critical,”  “marginal,”  and  “negligible”)  ranges  for  each  question/attribute.  Warfighters  should 
also  provide  a  “good  enough”  and  an  “optimal”  value  on  the  attribute  scale.  The  “good  enough” 
value  is  a  point  at  which  warfighters  assess  that  they  can  sufficiently  complete  their  mission 
without  significant  risk.  The  “optimal”  value  is  the  warfighter’s  ideal  operating  value  that  allows 
completion  of  the  mission  without  setbacks  or  hardships.  These  data  points  are  compared  with 
the  capability  requirements  where  applicable.  In  addition  to  asking  respondents  to  identify 
specific  values,  ask  for  brief  comments  about  why  they  chose  what  they  did.  These  comments 
can  offer  insights  that  make  the  analysis  more  defendable  in  the  end.  An  example  of  a  continuous 
scale  question  and  survey  format  is  in  Figure  2. 
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3  Definitions  for  the  severity  ranges  came  from  FM  5-19. 


Gap 

Gap 

Contribution 

Scenario 

Please 

S  One 

Participant  # 

Mobility 

Hard  Surface 
Speed  (MPH) 

Major  Combat  Operations  (MCO) 

Irregular  Warfare  (IW) 

Indicate  on  the  scale  the  points  at  which  hard  surface  speed  has  a  catastrophic, 
critical,  marginal,  and  negligible  negative  impact  on  the  currently  discussed  mission. 

75 


10 


15 


20 


25 


30 


35 


40 


45 


50 


55 


60 


70 


Indicate  on  the  scale  (★)  the  point  at  which  hard  surface  speed  (MPH)  is  "optimal"  for 
the  currently  discussed  mission.  Also  write  value  here _ 


Additional  Comments 

(Insight  into  "good  enough"  and  "optimal"  values  are  especially  useful) 


(Continue  on  back  if  necessary) 


Figure  2.  Continuous  Scale  Survey  Example. 


Not  all  attributes  can  be  evaluated  in  a  continuous  form.  These  are  best  answered  using  discrete 
values  (e.g.,  number  of  vehicles  in  a  platoon  losing  turret  firepower).  For  consistency,  use  the 
same  severity  categories  as  in  the  continuous  scales.  This  will  be  critical  when  assessing  across 
attributes  for  a  single  gap  and  when  assessing  across  multiple  gaps.  Unlike  continuous  scales, 
discrete  scales  may  not  warrant  a  “good  enough”  and  an  “optimal”  value.  However,  comments  or 
justifications  for  choices  are  still  highly  valuable.  Figure  3  shows  a  discrete  question  answered  in 
table  format. 
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Gap 

Gap  Contribution 

Scenario 

Please 

S  One 

Participant  # 

Complex 

Loss  of  Mobility 

Major  Combat  Operations  (MCO) 

Environment 

Irregular  Warfare  (IW) 

Indicate  on  the  table  (v^)  the  negative  impact  on  mission  (catastrophic,  critical, 
marginal,  or  negligible)  for  loss  of  mobility  in  1,  2,  3,  and  4  IFVs  in  a  platoon  for  the 
currently  discussed  mission.  You  must  have  only  one  S  per  row,  but  you  may  have  as 
many  v"s  per  column  as  you  like. 


#  of  IFVs  in  Platoon 
Suffering  Loss  of  Mobility 

Impact  on  Mission  (Please  ■/  One  Per  Row) 

Catastrophic 

Critical 

Marginal 

Negligible 

1  IFV 

2  IFVs 

3  IFVs 

4  IFVs 

Additional  Comments 


(Continue  on  back  if  necessary) 


Figure  3.  Discrete  Table  Example. 


Finally,  the  qualitative  gaps  require  assessment.  Survey  questions  for  the  warfighter  remain  the 
best  approach.  While  warfighters  will  be  unable  to  assign  a  number  that  represents  the  marginal 
mission  impact  of  not  having  enough  space  in  their  vehicle,  for  example,  they  can  describe  the 
impact  in  words.  Survey  questions  developed  for  these  gaps  are  best  answered  through  written 
responses  from  the  warfighters  for  collection  and  quantity-of-response  purposes.  Some  questions 
may  also  require  facilitated  discussion  with  warfighters  during  the  workshop.  When  using 
discussion-based  questions,  the  best  method  is  to  have  one  study  team  member  lead  the 
discussion  while  other  team  members  take  notes.  This  allows  the  leader  to  focus  on  asking 
follow-up  questions  or  to  keep  the  discussion  flowing.  An  example  of  a  qualitative  survey 
question  format  is  in  figure  4. 
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Gap 


Gap  Contribution 


Scenario 


Please 
^  One 


Participant  tt 


SWaP-C, 

Electrical 


Turret  System  Reboot 
Time  (minutes) 


Major  Combat  Operations  (MCO) 

Irregular  Warfare  (IW) 

o 


Describe  the  mission  impacts  of  incurring  a  turret  system  reboot, 
a  single  reboot  and  multiple  reboots  per  mission. 


Describe  in  terms  of 


Single  Reboot  per  mission  impacts: 


Multiple  Reboots  per  mission  impacts: 


(Continue  on  back  if  necessary) 

- 14 

Figure  4.  Qualitative  Survey  Question  Example. 


Workshop  Rehearsal. 

Once  all  inputs  are  developed  for  the  workshop,  select  representative  attendees  such  as  military 
functional  area  (FA)  49s  to  rehearse  workshop  execution.  Walk  through  the  timing  of  your 
schedule  and  have  your  representatives  ensure  the  survey  is  clear  and  answerable  before  you  go 
to  the  actual  workshop.  Getting  time  with  warfighters  can  be  rare,  so  make  the  most  of  your 
effort  through  practice  and  rehearsal. 

Workshop  Execution. 

Certain  “must  do”  actions  can  ensure  success.  Begin  by  setting  the  study  foundation.  This  is 
done  by  providing  the  background  of  the  study,  affirming  the  importance  of  warfighter  input,  and 
reviewing  the  scenario  briefings  to  establish  a  similar  operational  environment  for  assessment  of 
survey  questions  for  all  respondents.  Success  particularly  relies  on: 
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•  Communication  of  the  operational  environment. 

•  Effective  explanation  of  survey  sheets. 

•  Logical  ordering  of  survey  sheets. 

•  Verification  of  respondent  input. 

Communication  of  the  Operational  Environment. 

Establish  the  importance  of  the  surveys  to  the  future  of  the  Army.  This  emphasizes  that  the  time 
spent  at  the  workshop  is  valid.  Let  the  warfighters  know  that  their  voice  will  be  heard.  Highlight 
the  importance  of  the  workshop  by  effectively  describing  the  capability  gaps  and  the  operational 
environment  within  the  scenarios.  Ensure  the  key  features  of  the  scenario  are  emphasized, 
allowing  the  warfighters  to  appropriately  evaluate  the  capability  gaps. 

Effective  Explanation  of  Survey  Sheets. 

The  survey  sheets,  especially  those  with  a  continuous  scale,  are  neither  immediately  intuitive  nor 
overly  complicated.  An  effective  explanation  about  completion  is  crucial  to  establishing  a 
common  understanding  among  warfighters  of  the  survey  expectations.  Building  PowerPoint 
slides  showing  how  the  warfighters  will  fill  in  the  survey  sheets  is  effective.  Rehearsal  will  help 
ensure  proper  time  is  allotted  for  each  question.  If  time  becomes  an  issue,  ensure  that  valid 
results  are  your  priority  even  if  it  means  removing  some  questions. 

Logical  Ordering  of  Survey  Sheets. 

Surveys  should  be  grouped  for  ease  of  understanding.  They  should  first  be  grouped  according  to 
the  type  of  scale  involved.  All  continuous  scale  sheets  are  grouped  together,  followed  by  discrete 
scale  sheets,  and  finally  other  types  of  survey  questions,  such  as  the  open  responses  for  the 
qualitative  gaps.  The  purpose  is  to  keep  the  warfighters  accustomed  to  a  particular  method  for  an 
extended  time.  Surveys  should  then  be  ordered  according  to  warfighter  experience.  The  first 
sheets  should  contain  topics  with  which  warfighters  have  the  most  experience,  such  as  on-  and 
off-road  speed.  Topics  such  as  lethality  overmatch,  for  which  there  is  significantly  less 
experience,  should  be  put  toward  the  end.  The  purpose  is  to  build  warfighters’  confidence  in 
survey  completion  before  asking  them  to  think  more  critically. 

Verification  of  Respondent  Input. 

In  workshop  execution,  verifying  respondent  input  while  the  workshop  is  in  session  is  important. 
This  allows  the  study  team  to  discover  anomalies  in  responses  and  verify  that  the  respondent 
understood  the  questions.  For  example,  at  TRAC’s  GCV  workshop,  a  study  team  member  was 
dedicated  solely  to  data  entry,  with  the  focus  of  entering  all  participants’  scores  as  quickly  as 
possible.  As  the  data  were  entered,  preformatted  graphs  were  populated,  allowing  the  team  to 
instantaneously  visualize  results.  Once  the  data  were  entered,  the  team  did  a  quick  analysis  and 
produced  a  brief  back  to  the  participants  to  promote  group  discussion  and  gather  insights  into 
any  inconsistencies  in  the  data.  This  same-day  processing  proved  to  be  invaluable  to  the  team  by 
providing  an  opportunity  to  discuss  results  with  participants  while  the  results  were  still  fresh  in 
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their  minds.  Inter-rater  reliability  statistics  may  be  used  to  assess  concurrence  of  participant 
answers.  For  the  GCV  implementation,  graphics  were  used  to  visually  identify  variations.  In  the 
event  participants  stood  out  as  having  unusual  ranges  of  input,  the  study  team  discussed  their 
results  during  a  sidebar.  Refer  to  chapter  6’s  Survey  Validation  paragraph  for  some  examples  of 
outliers. 

Sensitivity  Analysis  Data  Collection. 

In  addition  to  gap  attribute  questions  during  the  workshop,  additional  information  to  support 
planned  sensitivities  may  require  data  collection  and  support  from  warfighters.  A  recommended 
sensitivity  is  for  warfighters  to  prioritize  the  gaps.  This  should  include  the  magnitude  of 
importance  for  each  capability  and  not  a  solely  ordinal  ranking.  One  method  involves  warfighters 
distributing  $100  among  the  capability  gaps,  giving  more  money  to  those  they  deem  most 
important  to  mitigate  and  no  money  for  those  they  think  are  nonissues.  The  priorities  are  used 
during  the  sensitivity  analysis.  See  chapter  5  for  a  discussion. 
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Chapter  4 
Gap  Analysis 


Purpose. 

Deriving  mission  impact  scales  based  on  warfighter  experience  is  critical  to  the  quantitative  gap 
assessment.  This  chapter  describes  the  development  of  those  scales  and  shows  how  data  are 
applied  to  the  scales  to  begin  the  gap  analysis. 

Qualitative  Gap  Analysis. 

Qualitative  gap  survey  results  require  analysis  to  determine  which  alternative  offers  the  best  gap 
mitigation.  The  analysis  of  qualitative  responses  requires  the  study  team  to  parse  survey 
responses  to  determine  trends  and  commonalities.  This  is  done  by  manually  reading  through 
responses  and  grouping  those  that  are  similar  in  nature.  Once  all  responses  have  been 
categorized,  the  study  team  can  record  those  responses  that  are  prolific,  with  attention  being  paid 
to  responses  that  suggest  a  feature,  a  capability,  or  a  quality  that  might  distinguish  among 
alternatives. 

Quantitative  Gap  Analysis. 

Analysis  of  the  quantitatively  assessed  gaps  is  more  structured  and  scientific  than  the  process  for 
qualitative  gaps.  The  first  step  is  to  create  a  database  of  survey  responses  for  each  question.  A 
simple  method  is  to  manually  enter  survey  responses  into  an  Excel  worksheet  and  use  a  new 
tab/worksheet  for  each  question  asked.  The  next  step  is  to  create  the  mission  impact  scales,  as 
discussed  in  the  following  sections  for  warfighter-assessed  capabilities  and  those  assessed  solely 
through  technical  data. 

Mission  Impact  Scales. 

Warfighter-assessed  Quantitative  Attributes. 

Several  steps  are  required  to  translate  the  results  of  the  surveys  into  a  single  scale.  Begin  by 
creating  histograms  for  each  survey  question  based  on  the  warfighter  responses.  Figure  5  shows 
an  example  of  a  histogram  for  the  dash-speed  survey  question.  Warfighters  agreed  on  the 
severity  of  the  mission  impact  at  some  points  but  disagreed  on  others,  as  the  overlapping  colors 
show.  This  poses  a  challenge  in  determining  at  what  point  along  the  scale  the  mission  impact 
changes  (e.g.,  at  what  point  does  “catastrophic”  become  “critical”?). 
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Figure  5.  Mission  Impact  Scale  Development. 

To  find  these  points,  fit  a  curve  to  the  data  for  each  severity  category  (“catastrophic,”  “critical,” 
“marginal,”  and  “negligible”).  Then  find  the  intersection  points  of  these  curves.  The  intersection 
points  will  identify  where  the  mission  impact  scale  changes.  To  do  this,  fit  the  data  for  each 
severity  category  to  an  appropriate  probability  density  function  (PDF)  using  a  distribution-fitting 
software  package  (a  detailed  discussion  is  in  appendix  A).  Note,  however,  that  when  fitting  the 
PDFs  to  the  data,  the  curves  will  keep  the  shape  of  the  histogram,  but  not  the  height.  Figure  6 
illustrates  this  height  difference. 
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The  red  “catastrophic”  severity  category  region  in  the  dash-speed  histogram  (figure  5)  has  a 
maximum  height  of  51.  The  highest  point  of  the  red  PDF  in  figure  6  is  0.0472.  This  height 
difference  must  be  corrected  to  find  the  accurate  transition  points  on  the  mission  impact  scale.  A 
transformation  scalar  must  be  found  to  adjust  the  PDFs  to  the  proper  height.  The  transformation 
scalar  is  calculated  using  the  equation  in  figure  7. 

max  height  of  histogram 

max  height  PDF  __  at  the  max  height  of  the  PDF  v_  (total  #  of  participants)  (max  height  PDF) 
X  ~~  total  #  of  participants  ~~  max  height  of  histogram 

at  the  max  height  of  the  PDF 


Figure  7.  Transformation  Scalar  Equation. 

As  previously  stated,  the  maximum  height  of  the  red  PDF  in  figure  6  is  0.0472.  At  the  PDF’s 
maximum  height,  the  histogram  shows  50  of  the  5 1  warfighters  agreed.  These  numbers  can  be 
put  into  the  equation  in  figure  8. 


0.0472  50  (5l)(0.0472) 

- = - »  X  =  - — — - >  X  =  0.0481 

X  51  50 


Figure  8.  Transformation  Scalar  Example. 


This  results  in  a  transformation  scalar  of  0.0481.  Finally,  divide  all  points  within  the  distribution 
by  their  respective  scalar.  Figure  9  shows  the  resulting  curves.  Each  curve  will  have  its  own 
transformation  scalar,  so  repeat  this  process  for  all  severity  categories. 


Once  the  true  intersection  points  are  found,  the  mission  impact  scale  in  figure  10  can  be 
produced.  Note  that  curve  fitting  instead  of  PDF  fitting  is  acceptable.  However,  fitting  the  PDFs 
can  be  useful  in  other  areas.  A  discussion  is  in  the  “Additional  Methods”  section  of  chapter  6. 
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“Good  enough” 


Dash 

Speed 

Sec  to  30  mph 


R  =  requirement 


Figure  10.  Mission  Impact  Scale. 


Additionally,  the  warfighter-provided  “good  enough”  values  are  averaged  to  determine  a  single 
“good  enough”  value  to  place  on  the  mission  impact  scale  (figure  10).  The  same  process  is 
completed  for  warfighter  “optimal”  values.  Also,  the  capability  requirement  is  placed  on  the 
mission  impact  scale.  This  requirement,  represented  as  “R”  in  the  figure,  comes  from  the  official 
capability  gap  documents,  most  typically  the  capability  development  document  (CDD)  or  ICD. 
As  discussed  in  chapter  3,  the  “good  enough”  and  “optimal”  values  are  compared  to  the 
requirement  as  a  validation  check.  When  inconsistency  exists  between  warfighters’  “good 
enough”  value  and  the  requirement,  alternatives  must  be  judged  against  the  requirement  value 
because  of  its  approval  in  official  documents.  The  resulting  mission  impact  scales  integrate 
science  and  experience  by  providing  the  basis  for  a  quantitative  assessment  grounded  in 
warfighter  expertise. 

Data  Application. 

The  next  step  in  quantitative  gap  analysis  is  data  application.  This  involves  applying  technical, 
performance,  or  operational  effectiveness  data  to  the  mission  impact  scales.  Data  application  for 
capability  gaps  being  assessed  quantitatively  is  fairly  straightforward.  Technical  experts  or 
warfighter  surveys  have  determined  the  mission  impact  scales,  and  the  study  team  must  now 
establish  where  each  alternative  being  considered  exists  on  the  mission  impact  scales.  To  do  this, 
use  technical/performance  data  to  assign  a  performance  value  to  each  alternative  and  then 
translate  that  value  to  the  mission  impact  scale.  For  example,  if  “alternative  1”  has  a  dash  speed 
of  28  seconds  to  reach  60  mph,  then  place  a  symbol  representing  “alternative  1”  on  the  mission 
impact  scale  for  dash  speed  at  28  seconds.  Repeat  this  process  for  all  alternatives  for  each 
mission  impact  scale. 


For  clarity  purposes,  once  data  on  the  alternatives  are  applied  to  the  mission  impact  scales,  they 
become  alternative  assessment  scales.  Figure  11  is  an  example.  The  study  team  should  also  keep 
the  gap/attribute  requirement  and  warfighter-defined  “good  enough”  and  “optimal”  values  on  the 
alternative  assessment  scale  for  analysis  purposes. 
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Dash  Speed 

Sec  to  30  mph 


60  50  40  30  20  10  0, 

Case  ^ Alt1  Alt2  Alt3  mAlt4 


GE-Good  Enough h»iLn»iolliiMl  from  Warfighter. 
*Cross  country  speed  data  represents  hilly,  dry  terrain. 


Figure  11.  Alternative  Assessment  Scale  Example. 


Gap  Attribute  Analysis. 

The  next  step  is  a  comparative  analysis  on  each  capability  attribute/gap.  The  combination  of 
warfighter  experience  and  scientifically  assessed  capabilities  makes  these  scales  powerful.  They 
may  be  used  to  assess  the  following  from  an  attribute  perspective: 

•  “How  much  capability  is  good  enough?”  can  be  assessed  by  the  location  of  the  negligible 
part  of  the  mission  impact  scale.  Trade  space  may  exist  if  the  alternative  is  on  the  far 
right  side  of  the  green/negligible  part. 

•  “Is  the  gap  attribute  mitigated?”  can  be  determined  by  the  alternatives  in  the 
green/negligible  part. 

•  “By  how  much?”  can  be  determined  by  the  distance  from  the  base  case  (presumed  to  be 
on  the  left  of  the  scale  in  red,  orange,  or  yellow)  to  the  location  of  the  alternatives 
(presumed  to  be  to  the  right  and  in  or  approaching  the  green). 

The  study  team  can  now  analyze  the  trade  space  and  attribute  mitigation  for  each  alternative. 
Figure  12  shows  a  sample  of  that  analysis. 
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IFV  Contribution  to  the  ABCT  Gap 

The  IFV  has  lost  the  ability  to  maneuveracross 
required  terrains  as  defined  by  the  ABCT  OMS/MP 
due  to  loss  of  powerto  weight  ratio  (increased  weight 
and  electrical  powerdemands),  reduced  ground 
clearance,  and  increased  physical  dimensions.  This 
hinders  the  unit’s  freedom  to  maneuverand  slows 
operational  maneuver,  increases  exposure  to  threat, 
and  reduces  combat  effectiveness  in  all  environments. 


Gap  Attributes:  Alternative  Results 

s SS  J,  »  i 

Secto30mph  -  r*  - 1 - 1 

60  50  40  30  20  10  0 

Cross 
Country 
Speed* 

Miles  perhour  q 

Hard 
Surface 
Speed 

Miles  per  hour  g  10  20  30  40 

Case  Alt1  Alt2  Alt3  mAlt4 

GE-Good  EnouahK>TIH«HTffiEn  from  Warfighter. 

‘Cross  country  speed  data  represents  hilly,  dry  terrain. 

■  NEG  MAR  ■  CRT  ■  CAT 


Gap  Mitigation  Insights 

•  Trade  space  exists  forthe  hard  surface  speed 
requirement.  However,  reduction  of  power  may 
affect  cross  country  and  dash  speed. 

•  Alt  3  and  Alt  4  completely  mitigate  the  mobility  gap 
forthe  IFV  and  provide  growth  forfuture 
capabilities. 

Moving  too  fast  has  negative  consequences  related  to 
target  acquisition,  ambush  susceptibility,  and  vehicle 
accidents.  -  Warfighter  3 

Alternative  Comparisons 

•  Warfighters  corroborated  the  CDD  mobility 
requirements  (R). 

•  Warfighters  prioritized  mobility  attributes  as 
follows:  1 )  dash  speed,  2)  cross  country,  3)  hard 
surface. 

•  The  Base  Case  and  the  Alt  1  perform  below 
Warfighter  “good  enough"  dash  speed. 

•  All  modification  variants  (*▲.'  )  perform  below 
the  Warfighter“good  enough1’ speed  for  cross 
country. 

•  All  alternatives  perform  within  the  negligible  risk 
range  for  hard  surface  speed. 


Alternative  data  provided  by  AMSAA. 

Requirements  (R)  from  30  August  2012  GCV  CDD. 


Figure  12.  Preliminary  Analysis  Example  (Mobility  Gap). 


Gap  Analysis. 

The  attribute  analysis  is  informative,  but  it  does  not  provide  a  means  to  answer  whether  the 
entire  gap  is  mitigated  when  gaps  are  composed  of  multiple  attributes.  To  determine  the 
mitigation  for  each  gap,  the  study  team  must  combine  the  attribute-specific  scales  into  a  single 
scale  for  each  gap.  Combining  the  alternative  assessment  scales  is  not  intuitive,  because  each 
scale  can  be  based  on  differing  metrics  (e.g.,  miles  per  hour  and  hours  of  operation). 

Normalizing  each  alternative  assessment  scale  to  a  common  metric  is  the  best  way  to  address  the 
differing  metrics.  The  GCV  study  team  based  its  normalization  on  this  scale: 

•  0-1,  “catastrophic” 

•  1-2,  “critical” 


•  2-3,  “marginal” 

•  3-4,  “negligible” 


This  process  assumes  that  the  transformation  follows  a  uniform  distribution. 


Normalization  allows  the  analytic  team  to  then  combine  all  gap  attribute  scales  related  to  the 
same  capability  gap  onto  a  single  gap  scale  by  averaging  the  sum  of  all  normalized  gap  attribute 
scores  for  each  alternative.  Examples  of  two  methods  for  displaying  the  normalized  gap 
assessment  scales  are  in  figure  13. 
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Alt  1 
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3.505 

3.604 
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Mobility  Scale 


Base  Case 

Altl 

Alt  2 

Alt  3 

Alt  4 

AVG 

Mobility 

(Total) 

2.373 

2.623 

2.890 

3.953 

3.792 

Figure  13.  Normalized  Gap  Analysis  Example. 


The  study  team  can  now  answer  the  same  three  questions  regarding  gap  mitigation  and  trade 
space  that  were  answered  for  gap  attribute  analysis: 

•  How  much  capability  is  good  enough? 

•  Is  the  gap  mitigated? 

•  If  so,  by  how  much? 


For  example,  figure  13  shows  that  all  alternatives  offer  improved  mobility  over  the  base  case. 
Alternatives  3  and  4  are  the  only  alternatives  that  have  a  negligible  impact  to  mission 
capabilities.  Both  fall  well  into  the  negligible  category,  meaning  they  both  offer  significant 
growth  potential  or  room  for  attribute  trade  space  as  discussed  in  the  Gap  Attribute  Analysis, 
section  of  this  chapter. 
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Chapter  5 

Overall  Assessment  and  Integration 


Purpose. 

Chapter  4  discussed  analyzing  single  gap  attributes  and  how  to  roll  up  and  analyze  a  single 
capability  gap.  The  next  logical  step  is  to  assess  alternatives  based  on  how  well  they  mitigate  the 
entire  set  of  capability  gaps. 

Overall  Gap  Mitigation  (Roll-up). 

This  overall  assessment  is  done  in  the  same  fashion  as  the  gap  analysis  explained  in  chapter  4. 
Each  individual  gap  is  already  on  a  normalized  scale,  so  that  step  is  complete.  Therefore,  the 
individual  gap  assessment  scales  in  chapter  4  must  be  combined  into  a  single,  overall  gap 
mitigation  scale. 

Following  the  process  described  above,  average  the  normalized  individual  gap  scores  for  a  single 
alternative.  The  result  will  be  the  overall  mitigation  score  for  the  alternative,  which  will  be 
placed  on  the  overall  gap  mitigation  scale.  Once  this  process  is  complete  for  every  alternative, 
the  overall  gap  mitigation  scale  will  be  complete.  The  study  team  will  then  be  able  to  report  out 
which  alternative  most  completely  mitigates  the  capability  gaps. 

Similar  to  the  discussion  in  chapter  4,  if  an  alternative’s  overall  mitigation  score  places  it 
drastically  to  the  right  of  the  negligible  region,  potential  may  exist  to  reduce  capabilities  while 
still  mitigating  the  gaps  at  hand.  This  may  allow  for  reduction  in  cost  or  shifting  of  money  into 
other  capabilities  whose  gaps  are  not  sufficiently  mitigated.  Go  back  and  analyze  the  attributes 
driving  the  overall  scale  to  the  right.  Figure  14  is  an  example  of  the  overall  gap  mitigation  scale: 


Figure  14.  Overall  Gap  Mitigation  Scale. 


Sensitivities. 

Even  though  the  supposed  “best  alternative”  has  been  determined,  sensitivity  analysis  should  be 
conducted  to  present  the  most  robust  analysis  to  decision  makers.  Some  sensitivities  to  consider: 

•  Examine  changes  in  gap  mitigation  based  on  changes  in  the  operational  environment  by 
prioritizing  the  gaps  based  on  their  occurrence  during  a  major  combat  operation  or  in 
irregular  warfare  environments.  Prioritization  may  be  accomplished  in  various  ways. 

-  Prioritize  the  capability  gaps,  1  to  n.  This  is  easy  for  the  analyst  to  incorporate  into 
the  gap  mitigation  workshop,  while  warfighters  are  comfortable  with  the 
requirements  and  the  terminology  of  each  gap.  Prioritization  data  take  little  time  to 
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collect.  Ranking  the  capability  gaps  will  not  show  how  much  better  one  gap  is  from 
another,  but  the  analyst  can  weight  effectively  using  the  rank  order  centroid  method. 

-  Prioritize  the  gaps  by  assigning  a  dollar  value.  With  the  instructions  of  allocating 
$100  to  fix  these  capability  gaps,  warfighters  intuitively  prioritize  the  gaps  and  give 
the  analyst  weights  as  well.  The  method  requires  more  quality  checking  to  make  sure 
all  dollars  are  used. 

-  Attributes  within  gaps  may  also  be  prioritized.  However,  incorporating  those  attribute 
priorities  with  the  gap  priorities  complicates  the  results  for  reporting. 

•  Use  frequencies  of  gap  conditions  in  a  similar  manner  as  the  priorities  described  above  to 
assess  gap  mitigation.  However,  frequencies  are  often  difficult  to  identify.  Refer  to 
chapter  6’s  Challenges  section  for  more  information  on  identifying  frequencies. 

•  Assess  the  significance  of  changes  to  the  risk  ranges  by  varying  the  location  of  the  risk 
band  for  each  comprehensive  mission  impact  scale.  This  investigates  how  close  an 
alternative  is  to  crossing  into  a  better  nominal  category  (i.e.,  “negligible,”  “marginal,”  or 
“critical”)  or  exceeding  a  very  close  alternative.  For  example:  increasing  vehicle 
protection  or  cross-country  speed  until  the  alternative  crosses  into  a  better  nominal 
category  or  exceeds  its  neighbor.  The  amount  of  change  is  noted,  cost  is  evaluated,  and 
both  can  be  reported  to  inform  leadership. 

•  Assess  the  significance  of  changes  (if  any)  in  “most  mitigating  alternatives”  based  on 
minor  changes  in  their  total  score.  If  two  alternatives  are  relatively  close  on  the 
comprehensive  mission  impact  scale,  performing  sensitivities  on  the  gap  assessment 
scales  to  determine  the  impacts  on  the  comprehensive  scale  may  show  what  would  have 
to  change  in  the  lesser  capable  alternative  to  make  it  exceed  the  other  alternative(s).  The 
same  could  be  done  on  the  mission  impact  scales  that  roll  up  into  the  gap  assessment 
scales.  At  this  level  of  detailed  sensitivities,  capability  trade  space  may  be  identified. 

Integration. 

Normally,  gap  assessments  are  completed  as  a  sub-element  of  a  larger  study,  such  as  an  AoA. 
Integrating  the  gap  assessment  results  with  the  rest  of  the  overarching  analysis  involves  two  key 
steps.  Step  one  looks  at  emerging  and  final  results  from  the  rest  of  the  study’s  analytic  efforts 
and  integrates  those  results  with  the  attribute  findings.  Step  two  looks  at  the  reporting  formats 
and  creates  a  consistent  method  of  presentation  that  does  not  conflict  with  the  other  study’s 
analytic  efforts. 

Step  One. 

For  both  quantitative  and  qualitative  gaps,  look  at  the  other  study  analyses  and  determine  what 
insights  from  those  efforts  help  further  explain  or  support  each  alternative  assessment  scale.  For 
example,  the  quantitative  attribute  represented  by  the  scale  may  be  performance-based,  such  as 
cross-country  speed.  However,  there  may  be  operational  impacts  related  to  that  attribute  that  are 
available  in  other  analyses.  Highlighting  the  operational  impacts  associated  with  the  attribute  is  a 
critical  insight  to  add  (and  source)  for  the  gap  analysis. 

Step  Two. 

Format  is  critical  when  integrating  the  gap  assessment  results  with  the  rest  of  the  study  effort. 
The  information  must  flow  without  conflicting  with  other  information.  This  gap  assessment 
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method  provides  information  for  each  gap  that,  when  analyzed,  may  be  presented  in  different 
ways  to  fit  the  flow.  Examples  of  methods  are: 


•  Visual  method.  Figure  13’s  visual  method  of  presenting  the  alternative  assessment  scales 
(by  attribute  or  by  gap)  and  the  overall  gap  mitigation  scales  (across  gaps)  is  effective 
because  one  glance  can  show  where  the  alternatives  fall  on  a  mission  severity  scale 
(mission  success  to  mission  failure),  how  they  compare  with  the  current  capability  (the 
base  case),  and  how  they  compare  with  one  another.  However,  the  color  schemes  used  for 
the  mission  severity  categories  may  conflict  with  other  color  scales  presenting  other 
sub-analyses  of  the  study. 

•  Tabular  method.  Using  figure  13’s  tabular  method,  values  instead  of  colors  may  be  used 
to  represent  the  mission  severity  scale  values  for  each  of  the  alternatives.  This  is 
discouraged,  because  the  scale  values  are  difficult  to  explain.  High-level  decision  makers 
prefer  seeing  values  that  represent  metrics  of  capabilities  rather  than  categories  of 
severity. 

•  Insights  method.  This  method  lists  the  analytic  insights  found  when  using  the  scales  to 
compare  the  alternatives  to  one  another,  to  the  baseline,  to  a  “good  enough”  capability  or 
existing  requirement,  or  to  an  objective  goal  for  that  gap.  See  figure  12  for  examples. 

•  Percent  improvement  method.  Figure  1  ’s  percentage  improvement  method  works  in  most 
cases.  Percent  improvements  are  calculated  as  the  improved  data  point  divided  by  the 
base  case  data  point.  Percent  improvements  may  be  calculated  for: 

-  warfighter-identified  “good  enough”  and/or  “optimal”  values. 

-  threshold  or  objective  CDD  requirements  values. 

-  study  alternatives. 

The  percent  improvement  presentation  method  may  also  be  used  to  identify  how  much 
improvement  certain  alternatives  are  above  the  “mission  success”  start  point.  This  will  help  in 
understanding  how  much  more  capability  is  achieved  over  the  potential  minimum  requirement 
for  mission  success  -  which  may  become  potential  trade  space. 


Table  1.  Percent  Improvement  Example. 


Gap 

PERCENT  "IMPROVEMENT" 

Base  Case 

Alt  1 

Alt  2 

Alt  3 

Alt  4 

Alt  1 

Alt  2 

Alt  3 

Alt  4 

Force  Protection  -  Avg  P(k)  (Smaller  is  better) 

0.182 

0.179 

0.089 

0.021 

0.02 

2% 

51% 

88% 

89% 

Vehicle  Survivability  -  Avg  P(k)  (Smaller  is  better) 

0.379 

0.376 

0.378 

0.19 

0.175 

1% 

0% 

50% 

54% 

Lethality  Overmatch  -  meters  (Bigger  is  better) 

(negative  shows  threat  outranges  alternative) 

-717 

-596 

-600 

-650 

-60 

17% 

16% 

9% 

92% 

Dash  Speed  -  seconds  to  reach  30  mph  (Smaller  is  better) 

33 

28 

21 

10 

11 

15% 

36% 

70% 

67% 

Cross  Country  Speed  -  miles  per  hour  (Bigger  is  better) 

18 

19 

20 

29 

28 

6% 

11% 

61% 

56% 

Results. 

Regardless  of  the  presentation  method  used  when  integrating  the  overall  study  results,  none  of 
the  visual  methods  identified  above  can  replace  the  real  analysis  required  in  gap  assessment. 
Developing  the  scales  and  placing  the  alternatives  appropriately  on  them  is  not  enough  to  be 
considered  gap  analysis.  The  alternative  assessment  scales  and  overall  gap  mitigation  scales  are 
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only  the  beginning  of  understanding  the  gap  mitigation  potential  of  the  alternatives  or  the  level 
of  capability  required  to  mitigate  capability  gaps.  This  gap  assessment  method  merely  assists  the 
analyst  in  garnering  greater  quantitative  and  qualitative  information  from  valid  sources  to 
analyze  the  gaps. 
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Chapter  6 
Conclusion 


Purpose. 

This  chapter  highlights  the  challenges  and  the  potential  problems  with  this  new  gap  assessment 
approach  and  reemphasizes  the  potential  uses. 

Challenges. 

The  following  paragraphs  relate  to  specific  chapters  or  paragraphs  previously  discussed. 

Gap  Sources. 

Finding  the  appropriate  gaps  for  study  is  not  as  easy  as  chapter  2  may  indicate.  Often,  due  to 
changes  in  authors  or  responsible  organizations,  gaps  documented  in  the  ICD  and  CDD  do  not 
match.  The  best  resolution  is  to  crosswalk  the  documented  gaps  with  those  most  recent  from 
ARCIC’s  CNA  process. 

Gap  Condition  Frequency. 

Gap  condition  frequencies  are  often  difficult  to  define.  As  documented  in  TRAC’s  Operational 
Risk  Analysis:  A  New  Approach  (TRAC-F-TR- 13-026),  identifying  frequencies  is  difficult.  The 
Operational  Risk  Analysis  report  identifies  the  following  obstacles: 

•  History  is  often  not  a  good  predictor.  According  to  Dr.  Nassim  Taleb’s  book,  The  Black 
Swan,  using  historical  data  to  determine  far-reaching,  future  events  is  often  very  inaccurate. 

•  Point  of  view  matters.  Warfighters  who  have  found  creative  ways  to  carry  out  missions  may 
find  the  frequency  of  a  capability  gap  to  be  zero  when  soldiers  who  experience  better 
capabilities  understand  that  the  current  capability  is  inadequate  and  the  capability  gap  is  seen 
as  occurring  more  often. 

•  Finding  qualified  experts  is  difficult.  Placing  quantitative  values  on  frequencies  depends  on 
the  experts  and  the  type  of  frequency  being  defined.  In  the  same  manner  used  in  this  report  to 
develop  survey  questions  that  the  warfighter  can  address  (see  chapter  3),  care  must  be  taken 
in  developing  frequency  surveys.  It  must  be  a  reasonable  question  the  experts  can  answer 
based  on  their  expertise. 

A  strength  of  this  new  gap  assessment  methodology,  versus  most  operational  risk  methods,  is  that 
frequency  may  be  treated  as  a  sensitivity  and  is  not  a  necessity  for  determining  gap  mitigation.  Refer 
to  chapter  5’s  Sensitivities  paragraph  for  a  description  of  using  frequencies. 

Gap  Approval. 

Ensure  the  source  gaps  are  approved  before  refinement,  if  possible,  by  the  SAG.  The  capability 
gaps  are  the  foundation  for  all  analysis  and  the  basis  by  which  the  Office  of  the  Secretary  of 
Defense  (OSD)  and  DA  will  assess  mitigation  and  make  critical  acquisition  decisions.  During  the 
GCV  AoA,  the  gaps  were  not  briefed  to  the  SAG  or  OSD/DA  action  officers  until  after  the  gap 
assessment  was  complete.  At  that  time,  OSD  representatives  stated  that  “the  only  approved  gaps 
are  those  in  the  ICD.”  As  a  result,  the  reporting  was  done  based  on  the  ICD  gaps  that  were  not 
refined.  The  gap  assessment  documentation  in  the  report  appendix  had  to  explain  the  use  of  the 
armored  brigade  combat  team  gaps  and  how  they  linked  back  to  the  “official”  ICD  gap 
statements. 
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Survey  Validation. 

An  interesting  result  of  the  survey  sheets,  especially  continuous  scale,  is  that  warfighters  may 
complete  the  risk  scales  opposite  to  what  the  study  team  is  expecting.  For  example,  a  respondent 
might  judge  higher  off-road  speeds  to  be  associated  with  higher  risk.  The  problem  is  that  the 
result  cannot  be  used  according  to  the  methodology.  In  these  cases,  the  study  team  should  point 
out  the  result  to  the  respondents  and  ask  whether  they  understand  the  question.  If  the  respondents 
remain  adamant  that  their  response  is  correct,  then  the  study  team  should  acknowledge  their 
point  of  view  and  designate  the  data  as  an  outlier.  Although  losing  a  data  point  because  it 
contradicts  the  CDD  requirements  is  disappointing,  insisting  that  warfighters  adhere  to  the  logic 
of  the  CDD  could  damage  their  confidence  in  the  workshop. 

Gap  Prioritization. 

The  GCV  AoA  prioritized  the  gaps  within  two  different  scenarios  -  a  major  combat  operation 
and  an  irregular  warfare  context.  In  hindsight,  the  preference  is  prioritization  across  any  or  all 
scenarios.  A  single  priority  across  all  scenarios  is  easier  to  perform  sensitivities  on  and  easier  to 
explain  to  decision  makers.  Because  the  prioritization  occurred  before  this  gap  assessment 
methodology  was  developed,  the  complexities  experienced  could  not  be  foreseen.  Keeping  all 
prioritizations  and  sensitivities  simple  allows  easy  explanations  of  gap  results. 

Technically  Defined  Scales. 

Not  all  gap  attributes  are  appropriate  for  warfighter-developed  scales  as  discussed  in  chapters  3 
and  4.  In  some  instances,  a  comparable  scale  may  come  from  technical  experts.  For  example, 
probability  of  kill  (Pk)  data  are  typically  available,  but  to  ask  the  warfighter  about  partial  losses 
of  personnel  or  vehicles  is  unreasonable.  The  performance  analysis  experts,  such  as  Army 
Materiel  Systems  Analysis  Activity  (AMSAA),  have  scales  that  equate  partial  losses  to  severity 
scales.  These  technical  scales  may  be  used  in  place  of  warfighter  scales,  if  they  are  comparable. 
Refer  to  table  2  for  an  example  of  technical  scale  and  mission  impact  scale  compatibility. 

Other  instances  exist  in  which  technical/operational  data  take  into  account  a  more  holistic  view, 
and  asking  warfighters  would  give  the  study  team  an  unnecessarily  narrow  assessment.  If  using 
purely  technical/operational  data,  the  technical  experts  should  provide  severity  categories  that  the 
study  team  can  translate  into  the  FM  5-19  severity  categories  used  in  warfighter  surveys.  These 
scales  are  integral  in  the  success  of  the  new  assessment  method.  If,  despite  measurability, 
warfighter-informed  or  technical  data  scales  are  not  definable,  analysts  are  forced  to  use  the 
qualitative  approach. 

Additional  Methods. 

Other  methods,  in  addition  to  PDF  fitting,  could  be  appropriate  to  find  the  transition  points  on 
the  mission  impact  scale.  Two  promising  methods  are  the  averaging  method  and  using  curve 
fitting  instead  of  PDF  fitting. 

The  averaging  method  involves  taking  the  average  of  the  overlapping  areas  on  the  survey 
response  histograms.  A  notional  example  is  in  figure  15.  However,  it  still  involves  finding  an 
intersection  or  transition  point,  using  the  areas  of  overlap. 
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Table  2.  Comparable  Technical  and  Mission  Impact  Scales. 


Technical  Scale 
Definition 

Technical  Scale 
Category 

Mission  Impact 
Scale  Category 

Mission  Impact 
Scale  Definition 

The  vehicle  sustains  little  to  no 
damage,  and  there  is  no  loss  of 
functionality  due  to  impact  from 
threat  munitions. 

No  loss  of  function. 

Negligible 

Mission  failure 
unlikely. 

The  vehicle  damage  results  in  a 
slightly  decreased  performance 
for  a  single  capability. 

Limited  loss  of 
function  or  slight 
decrease  in 
functionality. 

Marginal 

Mission  failure 
possible. 

The  vehicle  damage  results  in 
moderate  decrease  in  performance 
for  multiple  key  capabilities. 

Significant  loss  of 
functions  or  decrease 
in  functionality. 

Critical 

Mission  failure  likely. 

The  vehicle  damage  results  in 
complete,  or  nearly  complete,  loss 
of  function  for  one  or  more  key 
capabilities.  Vehicle  may  even  be 
considered  unrepairable. 

Severe  to  total  loss  of 
functions. 

Catastrophic 

Mission  failure  nearly 
certain. 

•Method:  Intersection  Point  = 


Sum[(Metric)  *  ( Frequency )] 
Sum(Frequency) 


•Notional  Example: 


Red  n  Orange  = 


(12  •  17)  +  (13  •  18)  +  (14  -  22)  +  (15  •  23)  +  (16  •  15)  +  (17  •  13) 
17  +  18+  22  +  23+  15+  13 


Red  n  Orange  «  14.37 


Figure  15.  The  Averaging  Method. 


Another  method,  as  mentioned  in  chapter  4,  is  curve  fitting  instead  of  PDF  fitting.  Similar  to  the 
PDF  method,  it  involves  finding  the  intersection  point  of  two  curves.  This  can  be  done  with  a 
curve-fitting  software  package. 


Both  methods  would  likely  be  appropriate  for  finding  the  transition  points  between  severity 
categories.  The  analyst,  however,  loses  the  flexibility  of  having  the  probability  distribution  for 
each  category  on  the  mission  impact  scale.  Recall  that  the  gap  attribute  scores  were  based  on  the 
assumption  of  uniformity.  The  team  could  benefit  from  exploring  the  use  of  cumulative 
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distribution  functions  (found  from  fitting  the  probability  distributions  in  chapter  4)  to  generate 
these  scores.  This  may  offer  a  more  precise  comparison  of  the  alternatives. 

Study  Integration. 

Integrating  the  gap  assessment  into  the  overarching  study  effort  begins  at  the  study  planning 
phase.  Consideration  should  be  made  then  regarding  the  severity  categories  (how  many  and  their 
definitions).  Those  categories  and  their  definitions  should  be  coordinated  across  the  overall  study 
effort.  If  this  is  not  done,  the  potential  exists  to  do  a  lot  of  work  but  then  be  unable  to  use  the 
alternative  assessment  scales  in  the  final  results  because  the  color  schemes  and  severity  category 
definitions  conflict  with  other  sub-analyses. 

Summary. 

As  the  Army  faces  more  fiscal  constraints,  Defense  and  Army  officials  will  continue  to  ask: 

•  Is  there  a  capability  gap? 

•  How  much  military  capability  is  good  enough? 

•  What  is  the  operational  impact  of  the  capability  gap? 

•  Is  the  capability  gap  mitigated?  If  so,  by  how  much? 

This  new  gap  assessment  approach  provides  a  foundation  to  address  these  questions  as  well  as: 

•  Corroborating  or  helping  refine  capability  requirements  by  comparing  the  threshold  and 
objective  requirements  with  the  warfighter-identified  “good  enough”  and  “optimal” 
values. 

•  Identifying  potential  trade  space  in  requirements  and  capability  attributes  by  looking  at 
differences  in  the  above  or  between  the  alternative’s  value  and  the  “good  enough”  values. 

•  Assessing  the  operational  impact  or  operational  risk  of  capability  gaps  and  their  potential 
solutions  by  comparing  the  base  case  and  alternative  values  to  the  defined  mission  impact 
categories  of  catastrophic  (red),  critical  (orange),  marginal  (yellow),  and  negligible 
(green). 


The  results  provide  an  opportunity  to  address  the  decision  maker  issues. 


Msn  Impact/  Op  Risk 
of  the  gap  (Basecase) 


10 

15 

20 

a  i  | 

^  1 

Potential  trade  space 


Mitigation  and  “how  much”  is  measured 
from  the  base  case  to  each  alternative. 


I  Threshold 

“Good 

Enough” 


Objective 1 
“Optimal” 


Alternatives 


• 

=  Base 

case 

▲ 

=  Alt  1 

O 

=  Alt  2 

■ 

=  Alt  3 

Figure  16.  Sample  Results  to  Address  Decision  Maker  Issues. 
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Appendix  A 
Software  and  Tools 


To  fit  the  data  to  probability  distributions  discussed  in  chapter  4,  the  study  team  used  EasyFit 
software  by  MathWave  Technologies.  Data  were  input  for  each  category  (“catastrophic,” 
“critical,”  “marginal,”  and  “negligible”)  on  the  mission  impact  scale  and  processed  with  the 
EasyFit  package.  Figure  A-l  shows  how  the  data  are  input  into  the  software  in  a  spreadsheet 
format. 


Once  data  are  entered,  the  software  will  fit  them  for  each  category  to  42  continuous  distributions 
or  8  discrete  distributions.  It  will  automatically  output  a  graph  for  each  distribution  and  allow  the 
user  to  look  at  several  distributions  on  one  graph  for  comparison. 


Figure  A-2  shows  a  sample  of  output  for  the  Johnson  System  Bounded  (SB)  probability 
distribution. 
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Figure  A-2.  EasyFit  Distribution  Output. 

In  addition  to  providing  visual  representations  of  the  distribution  fits,  EasyFit  performs  three 
goodness-of-fit  tests  (Kolmogorov-Smirnov,  Anderson-Darling,  and  Chi-Squared)  for  each 
distribution  so  the  most  appropriate  distribution  can  be  identified.  Figure  A- 3  shows  the  results 
of  these  tests  for  the  Johnson  SB  distribution  that  was  fit  in  figure  A-2. 
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Goodness  of  Fit  -  Details  [hide] 

Johnson  SB  [#30] 

Kolmogorov-Smirnov 

Sample  Size 
Statistic 

P-Value 

Rank 

1014 

0.0239 

0.60002 

2 

a 

0.2 

0.1 

0.05 

0.02 

0.01 

Critical  Value 

0.0337 

0.03841 

0.04265 

0.04767 

0.05116 

Reject? 

No 

No 

No 

No 

No 

Anderson-Darling 

Sample  Size 
Statistic 

Rank 

1014 

0.54764 

2 

a 

0.2 

0.1 

0.05 

0.02 

0.01 

Critical  Value 

1.3749 

1.9286 

2.5018 

3.2892 

3.9074 

Reject? 

No 

No 

No 

No 

No 

Chi-Squared 

Deg.  of  freedom 
Statistic 

P-Value 

Rank 

9 

11.222 

0.26081 

3 

a 

0.2 

0.1 

0.05 

0.02 

0.01 

Critical  Value 

12.242 

14.684 

16.919 

19.679 

21.666 

Reject? 

No 

No 

No 

No 

No 

Figure  A-3.  EasyFit  Goodness  of  Fit  Tests. 

Once  the  distribution  was  selected  for  each  category,  the  probability  density  functions  were 
extracted,  and  the  breakpoints  described  in  chapter  4  could  be  found.  More  information  on  the 
EasyFit  software  is  at  http://www.mathwave.com/easyfit-distribution-fitting.html. 
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Appendix  B 
Glossary 


ABCT 

Armored  Brigade  Combat  Team 

alt 

alternative 

AMSAA 

Army  Material  Systems  Analysis  Activity 

AoA 

analysis  of  alternatives 

ARCIC 

Army  Capabilities  Integration  Center 

avg 

average 

BN 

battalion 

cat 

catastrophic 

CDD 

capability  development  document 

CNA 

capability  needs  analysis 

CoE 

Center  of  Excellence 

CRM 

composite  risk  management 

crt 

critical 

DA 

Department  of  the  Army 

deg 

degree 

FA 

functional  area 

FM 

Field  Manual 

GCV 

Ground  Combat  Vehicle 

GE 

good  enough 

ICD 

initial  capabilities  document 

IFV 

Infantry  Fighting  Vehicle 

IW 

irregular  warfare 

JCIDS 

Joint  Capabilities  Integration  and  Development  System 

mar 

marginal 

max 

maximum 

MCO 

major  combat  operation 

mph 

miles  per  hour 

msn 

mission 

neg 

negligible 

OIF 

Operation  Iraqi  Freedom 

OMS/MP 

Operational  Mode  Summary  Mission  Profile 

OP 

optimal,  operational 

OSD 

Office  of  the  Secretary  of  Defense 

PDF 

probability  density  function 

Pk 

probability  of  kill 

R 

requirement 

SAG 

study  advisory  group 

SB 

system  bounded 

sec 

second 

STAR 

System  Threat  Assessment  Report 

SWA 

Southwest  Asia 

SWaP-C 

size,  weight  and  power  -  cooling 

TRAC 

U.S.  Army  TRADOC  Analysis  Center 

TRADOC 

U.S.  Army  Training  and  Doctrine  Command 
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